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unmeasured characteristics that themselves are related to the program 
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incorrect estimation of the treatment effect when assignment to 
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techniques that vere recently developed ^n the econometric analysis 
of labor and markets and applies these techniques in order to resolve 
the bias problem of conventional evaluation models. The selection 
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to the in^itutional setting of evaluation research and to its 
application in labor economies. The paper concludes with several 
caveats^about the proposed technique, (Author/HK) 
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ABSTRACT 

Selisccivicy bias arises in prograta evaluacion when che treatment Cor 

concrol) status of the subjects is related to unmeasured characteriscics 

that themselves are related to the program outcome under study. This 

* 

situation potentially leads to a misestintatioa of the treatment effect. 
In this paper we adopt techniques recently developed tn the econometric 
analysis o£ labor markets to the bias problem in a conventional evalua- 
tion model, A resolution of the problem 'emerges under assumptions that 
are reasonably general* References x6 the institutional setting ot 
evaluation research and to applic^ations in labor economies place the 
"selection bias'* problem in a broader conteKC. The paper concludes with 
several caveats about the proposed techni'que. 



Issues in the Analysis of Selectivity Bias 
1. KTRODUCTIOtJ 

Selectivity bi^s^ arises in program evaluations' when the treatment 
(or^'control) status of the subjects is related to unmeasured character- 
istics that themselves are related to the program outcome under study* 
The tera "bias" refers^to the potential n:if,estxmation of the etfect of 
the treatment (or program) on the outcome. Selectivity bias is a concern 

whenever the assignment to treatment and control groups is not random, 
conditional on what€fver observable explanatory variables, if any, are 
used in the analysis. So stated, it is clear that the issue of selection 
bias is pervasive in empirical research in^^conpmics because the assign- 
ment of observations to the different statuses defined by the predictor 

i. 

or e:colat^atory variables of interest are seldom* explicitly random. Thus, ; 
there should be a common ground in the methods used to analyze selectivity 

bias in program evaluation and econometrics, distinct but increasingly 

1 

overlapping fields. 

In this paper we adapt techniques dev^tlop^d in the econometric analysis 
of lab9r marVc-its— particularly by James J, Heckman (1974, 1976, 1978, 1979>, 
G. S,. Maddala (197&, 1978) and fladdala and Lung-fei Lee (1976)--to the 
bias problem in a conventional evaluation model* \ resolution of .the problem 
emerges un4er assump*:ions that are reasonably general* 

The ne:ct section of the paper provides an institutional background 
by describing developi?,ents in program evaluation and in applied labor 
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econonics thac deal vich and illusCrate the issue of seleccioa bias. Seccloa 
2 zhea' iatrodyce^ a simple aodel that iiiustraces formally the ^taci^ticai 
issues, notes condiclons when no bias exists and whea Che direc::ion of bias 
is known, aad poincs Co several misunderstaadiags ia the progran. evaluation 
literature* Seccioa 3 surveys several ecoaometric approaches ia labor 
ecoaomics that have attempted Co eliminate (or, eqyivaleatly , to quaatify) 
che bias- .SecCioif^T ^hich* is Che core of che paper, applies che aeu 
ecoaoneCric approach Co the conveacioaal evaluacioa model Co iadicate how 
the selectioa bias ia chis model may be resolved. 

1* Developiaeats ia Program EvaluaClon aad Labor EcoiioQics lavolvin^ 
Selectioa Bias ^ ^ , 

Evaluation research begaa vith che analysis of rather s^-aLl-scale 
■ > 

t projecrs, :nainly in nedicine, experi.TieataX aad social psychology, educational 
^ychology, and economics. In economics, the basic approach was beaefic- 
cost analysis, vhich was dif f *irer.t from the more direct applicatioa of 
statistics in the other disciplines, Siace the tiiid"1960s evalyatioa 
case^rch ha^ beea applied to large-scale govemmeatal prograns whose nain 

purpose "^as to improve the weXl-belr.g of a sizeable segmant of society,^ 

i - ' ' . ^ 

EvaliJatidn ^^'as not the main objective of the progr^itns, which l^.rgely^ 

e:<plair\s why random assignment to treatmeat and control groups was 

alaost never emplo/ed, Nevertheless, ex pose evaluations were publicly 

daaanded because the successes or failures of che programs were major 

political ioSiies, 

TJ.e current practice of program svaluatioa research by ecoaomi^ts is 
quite different from the ex ante evaluatioa of water resource projects and 
ths like, which t^ere the ;nainsta7 of trsdit iocial .benef it-co^t analysis,^ 

~ S ' ' 
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In ex post evaluations, 't;he first priority is to esfj'mate the quanti- 
tative effect o£ the program, which logically precedes the determination 
of whether the benefits exceed the costs. This priority^Ruts a greater 
emphasis on statistical models and has brought economic evaluation research 
closer to the approaches used in other disciplines. 

Today, evaluation research is a huge enterprise in applied social 
science. The Evaluation Research Society was organized in 1976, and the 
^' journal Evaluation began in 1973 with a grant from the National Institute 

1 of Mental Health. Coursers on the topic are offered in graduate schools oE 

many universities, often in recently established departments or centers of 
public administration and policy research. A tw-volume^ Handbook of 
Evaluation Research , intended as a text for graduate codrses, includes a 

bibliography of some 1500 items, almost all post-1960 and aimost all by 
2 

nonec ononis ts. Other books on evaluation^ filling several sl^elves in one's 

3 

bookcase, also tend to be recently published and written b> noneconomists. 

The volume Federal Program Evaluation contains an inventory of approxi- 
mately 1700 evaluation reports produced by and for 18 selected Federal 

. 

agencies, covering the period 1973-1975. Many were written by researchers 

'* * 

4 

outside the government, but all were related to agency programs. One may 
safely credit (or blame) the federal government, beginning with the Great 
Society programs of the Johnson administration, for making evaluation 

research the growth industry that it now is. 

\ 

The topic of our paper, selectivity bias, occupies a small part of the 
statistical methodology aspect of evaluation research, an aspect which is, 
itself, only a small part of the total field. Nevertheless, selectivity 
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bias can be given a broad interpretation, with implications and appli- 
cations extending beyond evaluation research. For example, selectivity 
bias can be viewed broadly as a version o£ specification error in' statist 
tical models in which behavioral outcomes are functions of "predictor" or 
"explanatory" variables. In this version, the outcome of a program is 
examined in the same way as an outcome of a controlled experiment in a 
laboratory or, at th&' ott/er end of the spectrum 6f research settings, as 
an outcome in a historical process measured with time-series data. There 
is a coEJimon framework used to measure the causal effect on the outcome of, 
respectively, the program, the laboratory treatment, or , the historical event* 

Selectivity concerns the presence of some characteristic of the treatm<>nt 
(or control) group that is associated both. with receipt of the treatment 
and also with the outcome so as to lead to a false attribution of causality 
regarding tre^itment and outcome. So stated, selectivity bias is a version 
\ of omitted-v^ariable bias, commonly analyzed under the rubric of 

sv^- fication error in econometric models. The unbiased measur^ent of 
causal effects is a broad and complex topic* As the eminent statistician 
Edi^rds Deming (1975) stated, "Evaluation is a study of causes*" 
In labor economics the issue of selection bias in evaluation models 
has been confronted directly in a number of recent studies. Training and 
education programs deal with individuals who are either selected for the 
program or who are self^selected. If this selection process is not fully 
known to the investigator, an unbiased measuife of the treatment may be 1^ 
unobtainable. For explicit attention to this problem, see the \ 
evaluation study of a government training program by Orley Ashenfelter 



8 



5 



(1973) and the evaluaricns of compensatory sducaciori programs by Irvin 
Garflnkel aad Icyard Graniich (1973) a:^d by 3uri 5* Barno^- and Glen G* 
Cain (1977). Regulatory programs usually deal vi^h firms, and a^airi 
ttiere is selection by the agencies (or self-selection) £or participation 
In the regulatory process. The apparent effects of the "program may be 
biased because'the outcoa:es r?ay reflect unmeasured preexisting charactsr- 
istics of the selected firms. Evaluations of anti-discrlaination and 
affirmative action programs are a case in point, and Hecleian zad 
t^enneth I.. Wolpin's (l976) study of the Office of Federal Contract Compliance 
of ths U.S. D^^eparttnent of Labor deals explicitly wiLh selection bias. 
.'Another example is the study by Robert S. Smith (1975) of the l-^pact on 
injuries of the ^'target industries program*' of the Occupational Safety and 
Health Act (see also the critique by Jack E* Triplett [1975])/ 

There are also well-knowT\ examples of the selection bias issue 
ir. other areas of labor economics, testifying to the pervasiveness of 
Lhe issue in empirical econoniic research* Attempts to measure the effect 
of uaions on wages are especially interesting because ths selection process 
is so vafied in several dimensions* The units oz observation may be 

industries^ occupations, firms^ or individuals. There is an ela'uent of 

1 

self-selection among individual workers, dependent in part on the worker's 
preferences for unionism; there is selection by union organisers^ dsperdent 
in part on the costs and benefits to the union of organizing the work place; 
and there is selection by employers through their hiring and personnel 
policies. Some twenty .years ago H. Gregg Levis (1959) disgussed these 
potential selection biases in e^camining the union wage affect. Several 
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racenc econoiriecric studies have accempted to model che seieccion of wrkers 
into union statiis in the course "of estimacin^ che union, wa^e effecc: 
Ashenfelter and George Jotinson (1972), Peter Schmidt and I^obert ?, Strauss 
(1976), Lee (1978), and Duane Leigh (1978) (see also ?andall J. Olsen 
tl97S»# ^ The intricacies in this area are illustrated by noting that the 
bias may differ across demographic or social groups in che population. 
.Jtshenfelter (1972) estimated a larger union effect for blacks than for ^^hites 
and Zvi Griliches (1976) estimated a relatively large union effect on the 
uages of young men, ages 17-27, . Does the selection process operate differ-; 
ently ainong sych groups of workers, and, if so, are differential biases a 
consequence? ^ 

Labor supply studies of L-omen have probably provided the ::iOst: explicit 
attention tc the selectivity pias issue in econo^iics* Kere the rnarkec 
uage of women is che analogu^ to the treatnent in evaluation studies, and 
the selectivity process affects labor force par ttctpacion and thus 6he 
observability of the market vage. special feature here is that the sample 
for which «i wage is ;n^^asured is truncated"no vage is measured for non- 

working ^^-oinen. In che typical prograjvi-evaluation design, the creacment 

t . 

f 

variable is measured for both the treatment and control groups. Heck?nan 
provided an early analysis of this problesi. (1974) and a useful r^iew 
article^('i9Z6)* Yoram Ben-Porath jCl973^), Retiben Gro^iau (1973, /l974}, and 
Leuis (1974) had earlier discussions of this application 6c selectivity 
bias, 2nd Paul Schult3 (1980) and Giora Kinoch C1930) heve contributed 
more recent vork. 
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Even those rare examples o: economic research vhich ase controlled 
e:^perinier,::3 random assi^nnencs ::o progran scacus— nanely^ the 

negacive inconie ta:< experiments— have been forced to confront selectiiviCy 
bias. The issij€ arises with respect to attrition, parcicalariy when the 
sabjeccs leave- the experiment for one of the exisc mg welfare progra*Tis. 
Analyses oc shese exparijnents have included atteir.pts at modeling the pro- 
cesses of attrition and of participation in welfare prograrns; Karold U, 
Watcs, Jon K. Peck, and Michael Taussig (1977), and Garfinkel (1977). 

Other studies in labor economics will be cited in Section 3 as 

i 

illustrative of specific ap|>roaches to dealing with selectivity bias^ -Our 
thesis that the issue of selectivity bias is^ important and pervasive in 
econometric research as :;ell as in program evaluation research 3houl<i be 
nonccn trovers ia 1 . 

2, A SI>£?LE: FORMl^LATION OF TKE EVALiJAtlON MODEL ' 

^'e focus on chskt part of an evaluation that seeks to_ measure the effect 
of the urogram a specific quantified outcome* Many aspects of a full 
evaluation are ignored in this narrow focus; the costs of the prograni, the 
dollar-equivalent value of tha outcome, the administration of the program, 
the equity issues involved in the distribution of benefits and costs, the 
correspondence between measured outcomes and political objectives'^ the 
question of multiple objectives^ and others. 

For simplicity's sake,, assume that the outcome, y, is liTiearly related 
to the traatment status, z (defined by p^^rticipation in the program), and to 
an unobservad variable, defined as the preprogram "true ability** to 
acHieve the outcome. A purer random term, £, co[nplete3 the equation. In 

U 
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this r.ypoc;i?5i»csl evaluation rioc^l, vich y syscsjiiaii r-eiated J^o 

aad there v:ould no need .for atiy ocher.conrral variables* Th<i 
evaltiator's Interest is in the effect of z on y, given true abiliE:y> and 
by assumption v comp.\eceiy measures that true ability. In (1) belou, 
is the ^trtie treatment effect: 

(i) ; y - <x Z ^ H^'^'tT* ^ 

But lathis equation is nonoperational because^w is unobserved (which, 
incid^ntallyj is why assigning it a unit coefficient is innocent). How may 

the eyaluator persuade an interested audience that the :neasured effect of z 

\ . ^ - ' I 

on y ^i^s free of an^f contamination from a correlation between z and Wp f 

given that v is not available as an explanatory variable? Random assign-/ 
ment to the z status is convincing iti prijiciple. But the integrity of 
randomisation Enay be coiaproraised in practice (by reliance on volunteers, by 
the absence of a dotible-blXnd design, by attrition), and in any event, 
aL-nost all programs deliberately use nonrandoni assigni-nents. 

There should be widespread agreer*ent among economists that' random 
assigniiients are not essential to the estiuiatlon of unbiased treatment 
effects.-'^ As have argued in earlier papers, tinbiasedness is attainable 
vhen the variables which determined the assigninenc ar\^*cnovsii , quantified > 
and included in the equation: Goldberger (1^72), Barnow (1973) > Cain 
(1975). Assume that an obser'/ed varia^)le^ t^ was used to deterir.ine assign- 
ment into the treatment group (z = 1) and the control group (z = 0). In 
gcnsralj t> which we refer to*. as the selection variable^ would be a score 
based on a composite of variables^ some of which -^"oulc be correlates of 
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abilicy, .Nevsrcheless, since : is che only svditcnatic ^ecer^lnaEi: of 
:reacr*ent scatur, z will csocure any cotr<iiacioa bc^iveon z ?.ad Vms, 
che observed z coilld replace che ur.cbserv'-sd w as the e:<planaE:ory variable 
in (1). In equation (2), 3^' vould be unbiased* chac is, ecual to :t: 

(2) y = 3^^ ^ 

f 

The use of either ^" or c as an e^xplanaCory variable, Chen, vrill free* 2 
from the contamination^-which leads to selectivity bias. 

''Modeling the selection process" is, o£ cQurse^ precisely yhac one 
claims CO do when specifying a multiple regression vhich holds constant 
those traits of the unit of observation that affect che outcome and that 
ace <55;^^3?Sc^d with the input variable of interest. A purely random deter- 
minant of assignment would be harml^^ss, where ''random" refers to a selection 
varicible"5:ich. as the flip of a coin — that do^s not affect the outcome^ 

Theory is supposed to tell us which selection variables are associated with 
OLttcornes- As examples; how do persons get "selected into" different 
educational actainrnent categories or into union membership, and hoE7 do the 
variablss fully describing this process relate to an outcome variable^ like 
e^rnings^ 

To illustrate briefly the selection model with an axaaple that has 
spacvTied an extended controversy^ assume 2 is participation in the Head 

Start compensatory education program, y "is the postprogram test scors ti., t 

is presumed to measure cognitive echievement, and z represents the caniiXy 

income of the children* Assume, fur-ther, chat those children cor whom t 

is below the poverty line (t ) are in the program (2 = 1) and those with 

valuer of t ebove c are excltjded (2 =0). Even though che correlacion 
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becve^n v and z and nhe 'correlacion benwe^a z and t ar^ boch hegaciv^, cbere 
is no bias ia 5, in riOciel'*(2) . Figure I iliustraces che ::iasni^:^:<i^ of 
in Che case that che prograii is beneficial. 

A laisunderscand ing-has arisen in the uses and incerpre^acion o£ chis 

model. In a seminal paper on ayaluation methodology, che psychologist 

Donald T. Canipbell (1969) suggested Chat this model, vhich he calls the 

"regression discontinuity" design, is severely restricted- But his reasons^ 

in fact/ do no^t apply. Campbell and Albert Eriebacher (1970) say that the 

model requires a random assignment among "ties" on the boundary line of 

participation (t^ in our example)? at^d that "we i^ould learn about the eftects 

of the program only for a narrow band .... ICe would wonder about its 

6 

ef £sctlvenes.s for tha most disadvantaged." But model (2) uses the full 
range of t; ties are inconse<tuenti4l> and therefore no randorai^^ation is 
needad; and the antire range, of values of t tjrovides potential 
information for the effectiveness of the program, even for nonlinear or^ 
interactive effects. Tigures 2A and 23 illustrate negative and positive 
interaccions between the treatinent and t. The nonlinear functional forms 
shoivTi are chosen to avoid both deleterious and explosive treatment effects 
vhen t becomes large; such nonlinearities, of course, vould have -to be 
specified in the model* 

Let usxsxaij^ine ne:ct the case where che selection process is not kno^Ti 
precisely^ in the sense that the available data do not permit quancif ication 
of t. ^Assume that there is available a vector x^, composed of variables 
that are correlated vith — that is, proxy for — ability, At che same 
ti"e-r-'>J include variables which enter t. The equation to be fitted is 
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Figure 1 



y, Poj^ctc^st 



^^=0, concrol 



t, Family Xacome 



Figure 1. N'oarandoni assignnteat: to treatment and coJUrol grot^ps , basecl 
on family ihcome, t. Dashed lines represent aoaobserved 
extrapolation of y, given t aad the treatmeat/coatrol 

^ status , , , ... 
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Figure 2A 
Megat:i\re z y t interaction 



Figure 2IJ ^ 
Positive z X t interaction 
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z ~ ly treatment 



2=0, control 



z ^ If treatment 



z = 0, control 



Figures 2A ^Jonrandom assignments to treatment and control groups b^sed 
and 2B ^ on family incotae^^ t, in exampl^es where the -treatment inter- 
acts with t* Dashed lines represent nbnobserved eictra-* 
polations of y, given t and the treatment/control -status • 
In Figure 2A the^ upper solid line, showing a negative 
interaction, is assumed to reach an asymptote rather than to 
decline indefinitely, thus avoiding a deleterious treatment * 
effect at higher t values* Similarly in Figure 2B, a positive 
treatment x selection interaction increases at a decrreasing 
rate and reaches asymptote, thus preventing the ^reatment^ 
^ effect trom exploding at higheir t values* ' / 
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(3) y => Vj_ z + 12 + £ * 

An estimate o£ iii this model will in general be biased for a* As 
discussed ia our earlier papers, the bias depends on the covariaace of z 
and w conditional on k and may be positive or negative. On this point a 
misuader standing arose in the well-known and often-cited paper by Campbell * 
and Erlebacher (1970) dealing with an evaluation of Head Start* They argued 
tha^ Che direction of bias could be inferred on the basis of the x, z 
correlation; 

How can one tell which direction a snatching bias vill 
take? Only by having evidence of the nature of the 
population difference which 'matching attempted tq 
overcome * * * * This underraatching (on true ability] 
'Showed up on the socioeconomic * status ratings subse- 
quently made [which showed a. negative correlation ^ 
between socioeconomic status and treatment status}. 

It is the last sentence in the quotation that gav^ r^ise to the misundar- 

standing* In the example discussed> x was Che socioeconomic status of 

the parents* Knowing that the.2t> ^ correlation is positive, one cannot 

infer that a negative x, z correlation biases down the Head Start (or r.) 

effect* tJhat determines the bias is the conditional covariance, C(z,w[x) 

8 

and nat the unconditional covariance, C(z,w)* 

Consider the general empirical model represented by equation (3) above, 
in which th^ investigator has measures of x* As one polar case, 3c coincides 
with w, and there would be no ^bias in estimating the treatment effect 
regardless of dtie selection process* Here; C(z,w) ?^ 0 bud C(2,w|2c) = 0 
because *£ coincides with w* This polar case is presumably hypothetical 
because v is unobserved* As another polar c^ie, \ coincides with d, and 



14 



there is again no bias, as was discussed above in connection ^*ith equation 
(2). Here again C(z,w)^# 0, but CCsjwjx) = 0 because selection on t (= k) 
means that the only relation between w and z is that which is induced by the 
relation between w and t (» ^) , a source of correlation which is fully 
' captured by A third polar case is that t, and therefore the z status, 
is random, like a coin flip. Herie again, C(z,w|x) = 0, and indeed C<z,w) = 0 
This l^ayes th^ Important case when w is unobserved and the selection 
nonrandom and not fully measured. With no other out^side ir.iormation, 
nothing can be said about the direction of bias, ^^ith information about the 
^nditional covariaaces — in particular C(z,w|x), but also C(t,xlw) or its 
^qi^valent in sign, C(z,3c|w) — direction of bias may be 4etermiaed, With 
iarormatioa about the distribution of the unobserved variables, t and w, 

f 

aloag.with information about the functional form relating t and w to jc, it - 
turns out—perhaps surprisingly—ctiat the bias may be quantified, and 
further, that this achievement does not require information about the signs- 
of the relations, conditional or unconditional, among t, w, and ^, beyond 
what may b$ ^lir^ctly measured on'tfie basis of the observed values of z and ic. 
The quantification (or equivalently the elimination) of bias in terms of the 
model of evaluation is demonstrated in Section 4* Let us first survey 
briefly the handling of selectivity bias in the recent econometric studies 
in labor economics. 

3. APPROACHES TO SELECTION BIAS 'ltl LABOR ECONOtlETRICS 

As we read the labor .econometrics literature, it taps additional infor- 
niation about the selection process to obtain an-unbiased estimate of the 
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treatment: effect vhen the observed x-variabLas exhaust neitiher t nor w. In 
essence> a selection equation with z as the dependent variable is speci:iec> 
and restrictions are [placed on it relative to the outcome equation— our (3). 
The role of the restrictions is to purge the apparent treatment effect— 

in our aquatioh (3)~of any preexisting differences between the treat- 
ment and control groups. In our reading> we detect: two types ol.restric- 
tions. The first dype s^^ifi<is that one or more* variables determining 
selection do not affect the outcome and hence are excludable from the out- 
come equation. Thus, information on variables excluded from x is required. 
The second type specifies c functional form for th.e delation between V and- 
w and a nonlinear relation Jbetween z and This leads to a nonlinear \ 
function of in the outcome equation, which serves to control for any 2, 
w relation that is net of x^- ■ ^ . ^ 

In distinguishing these two- types of restrictions, we pass over what 
is no doubt the most common approach; simply assume away the selection bias 

. / /' - 

after ^ diligent .attempt Co include a large number of variables in x that 

control for abilitv, w. The' argument , or assumption, is that whatever the 

selection variables may be, beyond those included in x, they are unrelated 
to outcome. In the two-equation recursive formulation for z and for y,_ . 

which we discussed above and present formally In section 4, this amounts 

to assuming that 2, is determined by berth 5c\and a disturbance that is uncorre- 

lated with the .disturbance, equatiotj (3) determining y. Most of the 
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research estimating the effect of education Ythe treatment variable) on 
earnings (the outcome) adopts this approach. Also> the empirical estimates 
of the Head Start ef^^ct on test scores by Barnow and Cain Cl977) fell In 
ttils category, 

\ 

\ 



16 

Two examples suffice to illuscrate the first type of restriction, 
which is to obtain additional x's that have no effect on the'outcome and 
then to jse this information to identify the treatment effect. A 
trivial but universally accepted case is the coin flip. When the t 
variable is generated naturally by environmental or market forces^ 
however, the assumption that the '^selection variables" have no effect 
on the outcome will usually be controversial. For example^ in a study of 
the effect of unionism on wage rates^ Ashenfelter and Johnson (1972) 
variously assumed that the concentration ratio of the industry or the 
region of residence (South or non*South) determines the extent of 
unionism, while concentration and region have no causal effect upon 
wage rates. Thus> those two variables are excluded from the structural 
equation for wage's* The authors' suggested that those identifying 
restrictions might be considered arbitrary. 

The secori'd type of restriction permits distinguishing between the 
way the x-variables affect selection fVom che way they affect outcomes 
(or, equivalently, the ^ay they represent ability), David Greenberg ai:td 
I Marvin Kosters, (1973) analyzed the relation between labor supply (an 
outcome) and nonlabor income (the treatment). Their initial empirical 
work led them to specify a selection process that involved differential 
tastes or preferences for asset accumulation (and, thus, tor nonlabor 
income). They used some of the x-variables from the labor supply equation 
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to estimate asset preferences by the device of a regression of observed 
assets on the selected 2i"vairiables , Predicted asset;; were included in the 
final labor supply equation, and identification of Its effect was achieved 
by virtue of its nonlinear relation to The controversial feature of 
this procedure is that the specification of the nonlinear function between 
observed assets and the set of 3c-variables was somewhat ad hoc. 

More recent work, beginning with Heckman (1974), appears to be consider- 
ably more general, A nonlinear functional form in x that serves to identify 
the treatment effect is not Imposed directly but rather emerges from two 
assumptions^ one, that the distribution of the error terms In the eqiiations 
for the omitted variables, w and t, is bivarlate normal; and, two, that the 
functional forms relating v an"^^ t to are known — in practice, known to 
be linear. We develop this in Section 4., 

Heckman's model, dealing.vith^^the labor supply , of women, is compli- . 
cated by two features: one,,^.lraultaneity, which does not appear in the" 
typical evaluation model; arid the other, truncation, which does appear_but 
in a somewhat disguised form* First, in the women's labor supply mjodel, the 
analogue to the oinltted ability variable is the "shadow price of time'' in 
housework, and this is assumed to be affected by the outcome variable, 
hours of market work. Thus, a simultaneous-equation model is specified and, " 
as with the Ashenfelter and Johnson model, additional identifying restrictions 
are required* Second, as mentioned earlier, the analogue to the treatment 
variable Is the wage rate, and this Is not observed for women who supply 
zero hours/of market work. Thus, the distribution^ of Wage rates is truncated 
and, unlike the treatment variable in program evaluation models, is npt 
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measured £or all observations. This is only an apparent difference, however 
In the evaluation model with differential ability between treatment and 
control groups, Che two groups are effectively truncated (or censored) on 
ability, and the technique for correcting for the omitted variable bias is 
the same as the technique for correcting for the truncation bias. 

4. UNBIASED ESTIMATIOH OF TMiATi^tENT EFFECTS ,IN THE EVALUATION MODEL 

I ' 

We now sharpen the specification of the evaluation model in a way which 

^ '■ 
'J 

will permit unbiased estimation of the treatment effect* The observable 
variables are y = outcome, z = treatment (2 = 1 for treatment group, 2 0 
for. control group), and x = vector ot covariates (including the constant). 
The unobserved variables are w ~ ability and.c = selection, along with 
various disturbances to be introduced below. By definition of t, assign- 
ment Co the two groups 'is determined by 

ii if t > 0 
0 if t < 0. 
By definition of w, outcome is determined by 

I 

(5). y^w + az + e, 

o 

where the disturbance is normally distributed, independent of w and 2, 
with expectation ^ero and variance cf^q* 

Consider the joint probability distribution of w, c, and x in the ' 
initial population, that is, prior to selection and treatment. We suppose 
that 



(6) w = e| x + 
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(7) ' t = 5^ x ^ 

where and 5,2 coefficient vectors ^nd ^iiere the disturbances 
and ^2 bivarla'ce-normal , independent of 21 (^^^ -q^ ^^^^ expectations 
zero, variances and cbvariance c^^. Note that in (6)-(7) 

two sources of nonrandom selection are distlnsuished: u and t can be 
correlated via their common dependence on x and via the correlation of 
tlieir disturbances and Equations (6)-(7) are not intended to X>e 

causal: ue need not say that x determines w and/or t< Kather, che'speci- 
fication may be interpreted as purely descriptive of the joint probability 
distribution of w, t, and 

Proceeding to the analysis, we substitute (6) into, (5) to get the 
outcome equation: 

where = + E . :iov and are bivariate-normal, independent of 

X, with eic'pectations zevo, variances " ^11 *^oo* covariance 

^23 ''**^12' ^'^ seek E(y|ic, z), the e:;pectation of outcome conditional on the 

.1 

covariates^and the treatment dummy. 

From <4) and^ (7) we se^ that the event 2 = 1 is equivalent to the 

event 9^2 +'"^2 ^ ^* ^2 ^ " -2 ^' ^^^^^^2^^2^ ^ ' I'i' 

where, a^'^ and 8_ (l/o^) 82^ By the saine argument, the event 2 = 0 

is equivalent to (£2/02) ±^ ^)ow ^2^^2 ^ standard norr al variable 

independent of Since 2 is binary, it follows that 

(9) - ECzjx) - Prob {z = Ijx) = 1 - F(- 9.'x) = F(6'x), 
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whi&re F(') denotes che scandaid normal cumulacive disciributioii fanccion. 
Further, ic follows thac 

■ • \ 

(10) Uit^/O^)}^, z = 1) = f(|;x)/?(3'x), 

(11) £((£2/02) !x, 2 = 0) = -f(a'x)/(i - F(i'ii)), 

where f(*) denoces che scaitdacd itoriaal densicy funccioit; see Johnson and , 
:<0C2 (1970, p. 81). Using f and F as shorthand for {{^'k) and_EL(a^x) 
res?eccively> we can assemble (lO)-(ll) 'iitco> say, 

E((€2/a2)lx, 2) = 2-f/F (1 - 2)f/(l - F) 

(12) ' = h(x, z; I). 

Then E(e^jK> z) =^ h(x, z; 6^, and wich che discribuCi^nal 
intorr:iatioii under (8), ic follows thac 



where ;j a.^/o^: see Johnson and Kdtz (1972, p. 112). 

In view of (13), che expectacipn of (S) condicional on x and z is 

(14) E(yjx, 2) = l{x + z + p h(x, 5:;\9)- '^^ 

.Sirjce this is « condicional Gxp^ccation function relacing observable variables, 
its paraniecersj namely 6^^, a, and 9^, are consistently estimable by nonlinear 
lease squares. We have thus escablishGd a method of obtaining an unbiased^ 



21 



or> CO De precise, a consiscenc escimate of che creacnient: effect la our 
specification of the evaluation model* In doing so, ve I'vave si:r.ply restat^<t 
the argumeats in Hecknan (1976)' and Maddala and Lee (1976).^ ■ 

To estimate "(14) in practice> a cwo-step t^rocedure raay be used. rirsc> 
estimate ^ by niaximum-likelihood probic analysis of z on and insert chose 
estimates £ in^Jllac^D of £ in (12) co calculate h = h(Ky z^, £) at each obser- 
vation. Sec6h3> e&timace^T > ci> .and ;: by linear lease stjuares regresaifOn of - 
y on x> Zy and h. This too provides consistent estimates. 

A main theme in che evaluation research literature on selection bias 
is thus verified: linear regi^ession of y on x and z produces biased estimates 
of a, che treatment effect. But in the present formulation the precise 
source of the bias is apparent, namely oiaission of the h(^> z; ^ variable 
in (14). Ottce this tern is included, least-squares regression gives a proper 
estinacs of a* We observe from (lA) that linear regression o: x and z alone 
would gtve^an unbiased estimate of a in tbe special case u = 0, that is 
^12 \^^^ ^^^'> ^liO/"" ^ equivalently> C(w, zjx) = 0, This verifies 
che resultsWiscussed informally in sections 2 concerning the absence of bias 

when assignment is purely random, or purely on the basis of the observable 
lA 

covariates. ^ 

^ still simpler approach' is also available* Vdth the aid of (9) ar.d 
(12) we recognize that conditional on x^^ h(}^y z\ 9^) is a constant ti::ies 
z - Z(z\^ > so that 

(15) E[h(x> z; 5)lx] - 0. 

Applying che iterated expectation rule to (14) then givss 
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(16) Z{y\x) = E ai/'^y 2>) = ^ ^Alizh) = i^ic ^ .-irie'^), 

ladeedj (16) can be obtained directly from (8): see Maddala aad Lee (1976, 
p. 528) aad thus holds without assumiag ciomality for ilow (16) is 

also a conditional expectation fuciction relating observable variables, so 
its paraiieters^ a> and 9^ are consistently estimated by nonlinear lease 

squares, in practice a tvo-step procedure nay be used: First, estimate 

- ^_ \ 

9^ by maximum-likelihood probit analysis ol z on ^t^d insert those estitTiates 
in place of 9 to calculate z » F(9_*2l) at each observation. Second^ estimate 
9. and Ci by linear least squares regression of y on x and 2, From this 
perspective, linear regression of y on k and 2 produces biased estimates of 
a because the variable z has not been purged of its endogeneity. 

For more formal discussion of these and alternative estimation procedures 
see also Takeshi Amemiya (1^78), Heckinan Cl^'8) and Lee (1979), 

We believe that this straightforward application of the Keckman- ' 
tladdala-Lee approach ^resolves in principle the problem of selectivity bias 
as it arose in evaluation research. Having reached that point, ve nest 
indicate that a number of serious probleias require attention ar^ong which 
are the following: 

(i) Choice among alternative consistent estimation procedures, 
' Cii) High degree of collinearity in the second-step regressions- 
Ciii) Robustness of e^stxjiators to non-normality of disturbances; see 
Crawford (1979), ' ^ - 

(iv) Missoecif ication of original models in %'hich case th^^ nonlinear 

terms h(2v, z\ 5) and F(2*ii) ^^V be pro>;ying for omitted variables and/or 

'I 

nonllntiarities, 

(v) Multiple selection rules; see Kaidman (1979)- " — 

I 
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This llsclng Includes both conceptual problems—parcicularly (v) , 
(Iv), and possibly (lli)— and problems of implenencaclon— mainly (i), 
(ii)^ and (lii). We are noc able at chis cime co assess che frequency 
or seriousness of these problems, so the list will have co scand by 
lcself^as_.a-racher stark agenda for fucura analysis. 
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MOTES 

'Hw earl:? laflueatial books ia chls craditloa are J^oland N, Mc:<eaa, 
Efficleacj ia gov'ernmeat: chrdugh systems aaalysia 0*ew York: John, '^ilevj 
1958), aad Qcco Eckscela> tracer resources developmeac : The ecoaumlcs of 
project evaluacJLoa (Cambridge, Mass,: Harvard Ualversity Press, 1958) * 

2 

E, L. Scruealag, aad Marcla Gucceacag, eds,, Haadbook of evaluatioa 
research ^ 2 vols., (Beverly Hills, Calif., 1975, Sage Publicacioas) 
These volumes will be ciced as HER I aad Her II . 

3 

See Che appeadi:^. 

/ 

4 

Federal program evaluatioat a directory for che Coagress , by the 
U.S* Geaeral Accouaclag Office (Washlagtoa, D,C.; GPO, 1976). Mo 
iaformacioa is givea oh che cose of these evaluatidas, , Perhaps a 
coaservatlve esciinate of the average cost of aa evaiuacioa study is 
$50,000. This implies chac $85 millioa was speac ia che 1973-75 period. 

^However, coasider the followiag scacemeac by che ecoaomist Alice 
Rivlia C1971, pp. 111-112): 

A valid experimeac requires that iadividuals be 
assigned co creatmeac or coacrol groups by a raadom 
seleccioa process, Chaace must eacer. ^ govera- 
' meat official may fiad ic far more difficulc Co 
eKplaia co che public chat he is allocaciag a scarce 
resource oa che basis of chaace thaa co defead some 
other selectioa cricetioa such as aeed or merit or 
"first come, firsc serve," But if such criteria 
are used, those aot selected caaaot validly be com- 
pared with che creacmeac group co escablish the 
ef f ecciveaess of the creatmeat, because the two 
groups may differ ia imporcaac ways^ 

^Campbell (1969) is repriaced ia H£H I , pp, 71-99. Campbell aad 
Erlebacher Cl970) is repriated ia HER I * pp. 597-617. The quotation ia 
the te:<t is oa p. 615 of HER 

^ HER I , p, 606. 
8 

Campbell aad Erlebacher also referred to the fact that the aaalysts in 
che study being criticized had a more difficult time fiadiag the "most 
diiiadvaataged" childrea amoag the coatrol group (HER I, p. 607). This 
proves aothiag, however. The most disadvaataged childrea may be more 
difficult CO recruit generally for either treacmeat or coatrol groups. 
Or the most disadvaataged coatrol childrea who were fouad nay have beea 
lower oa true abiliiiy thaa the treatmeac childrea of the same measured 
status. 
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' ^Accuallv their tnodel was :iiore elaborate thaa our two-equatioa 

(saleccion) status and wage rates (outcoraes). Thus, they requires ^ 
ideatJfJia restrictions \o estimate the effect ot wage rates oa uaioa 
status aad used the restrictioas that aa industry s average educational 
level aad percentage of Eemale employees affect wage rates but aot the 
extent of uaioaism. Clearly, these are also debatable -assuinptioas. 

a„d Woodward (1977), Porter aad ChlSuoos (.1975), Rubla "''/'"S.iy 
(1978) IC now appears to us thac these attempts all raa astray precisely 

lu^iit^.T/olXTi^ =^-r;"%'ai'^a^nraL^s™i::ra9?i)-. 

stimulated by so.e Ideas ia ; ; e "„o « a"! spe=i£i- 

T»* Is worth nociag cnac cne. present ayK"-"**^^* ,t 
cLioa of Jhe dlrlctioa of select iotr-"creamW ^^/-//^■;|;,,,f3 
direction caa be inferred ex post from Lhe sigas of the coeEEicieats. 
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